Papers with video-language models
LAVIS: A One-stop Library for Language-Vision Intelligence (2023.acl-demo)
Copied to clipboard
| Challenge: | a new open-source library for language-vision research and applications is available for free. |
| Approach: | They introduce LAVIS, an open-source deep learning library for LAnguage-VISion research and applications. |
| Outcome: | The proposed library is open-source and highly extensible and configurable. |
Grafting Pre-trained Models for Multimodal Headline Generation (2022.emnlp-industry)
Copied to clipboard
| Challenge: | Existing approaches to generate video headlines with pre-trained language models are labor intensive and impractical. |
| Approach: | They propose to graft the encoder from the pre-trained video-language model on the generative pre-trainer model and propose a consensus fusion mechanism for the integration of different components. |
| Outcome: | The proposed model achieves strong results on a brand-new dataset collected from real-world applications. |
Contrastive Video-Language Learning with Fine-grained Frame Sampling (2022.aacl-main)
Copied to clipboard
| Challenge: | despite recent progress in video and language representation learning, the weak or sparse correspondence between the two modalities remains a bottleneck. |
| Approach: | They propose a fine-grained contrastive objective for video frame sampling to improve cross-modal correspondence. |
| Outcome: | The proposed approach achieves state-of-the-art performance on YouCookII with long videos. |
MovieCORE: COgnitive REasoning in Movies (2025.emnlp-main)
Copied to clipboard
Gueter Josmy Faure, Min-Hung Chen, Jia-Fong Yeh, Ying Cheng, Hung-Ting Su, Yung-Hao Tang, Shang-Hong Lai, Winston H. Hsu
| Challenge: | MovieCORE is a video question answering dataset that focuses on surface-level comprehension. |
| Approach: | They propose a video question-answer dataset that uses large language models as thought agents to generate and refine high-quality question-anchor pairs. |
| Outcome: | The proposed model improves model reasoning capabilities post-training by 25% . the proposed model is based on a large language model and is scalable to a wide range of tasks . |
Multi-modal Action Chain Abductive Reasoning (2023.acl-long)
Copied to clipboard
Mengze Li, Tianbao Wang, Jiahe Xu, Kairong Han, Shengyu Zhang, Zhou Zhao, Jiaxu Miao, Wenqiao Zhang, Shiliang Pu, Fei Wu
| Challenge: | Existing models for Abductive Reasoning are limited in their ability to infer the most plausible explanation of incomplete known phenomena. |
| Approach: | They propose a vision-language task that aims to imagine the most plausible event by spatio-temporal grounding in past video and infer the hypothesis of subsequent action chain layer by layer. |
| Outcome: | The proposed model outperforms existing video-language models in terms of effectiveness on the proposed dataset. |
UNICORN: A Unified Causal Video-Oriented Language-Modeling Framework for Temporal Video-Language Tasks (2024.emnlp-main)
Copied to clipboard
Yuanhao Xiong, Yixin Nie, Haotian Liu, Boxin Wang, Jun Chen, Rong Jin, Cho-Jui Hsieh, Lorenzo Torresani, Jie Lei
| Challenge: | Recent advances in large multimodal models have encouraged the development of large multi-modal models . however, it is unclear how to extend these models to the more complex video domain . |
| Approach: | They propose a visual instruction tuning framework to address temporal video-language tasks . they collect a dataset and fine-tune the framework on instruction-following data . |
| Outcome: | The proposed model can perform better on established temporal video-language tasks without training objectives and intensive pre-training. |
ViPE: Visual Perception in Parameter Space for Efficient Video-Language Understanding (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing video-language models rely on concatenating visual tokens with textual inputs for joint modeling, but this method suffers from significant inefficiency when scaling to long videos with dense visual inputs. |
| Approach: | They propose a video-to-parameter efficiency paradigm called ViPE that transforms video content into visual perceptual weights, which are directly injected into the LLM’s parameters. |
| Outcome: | The proposed model reduces FLOPs by 85% and inference time by up to 65% while reducing FLOP and FLOP inference times by up-to-65%. |
Deep Temporal Reasoning in Video Language Models: A Cross-Linguistic Evaluation of Action Duration and Completion through Perfect Times (2025.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that video-language models struggle to mirror human-like temporal reasoning grounded in video . Sequential events are not simply arranged chronologically; rather, one event triggers the next upon reaching its completion. |
| Approach: | They propose a quadrilingual dataset to assess temporal reasoning in video-language models . they pair everyday activity videos with event completion labels and perfectivity distractors . |
| Outcome: | The perfect times dataset examines whether video-language models comprehend temporal dynamics . it combines everyday activity videos with event completion labels and perfectivity distractors . results show that state-of-the-art models struggle to mirror human-like temporal reasoning . |
TV-TREES: Multimodal Entailment Trees for Neuro-Symbolic Video Reasoning (2024.emnlp-main)
Copied to clipboard
| Challenge: | TV-TREES is the first multimodal entailment tree generator for video understanding . it searches for trees of enanglement relationships between text-video evidence and higher-level conclusions that prove question-answer pairs. |
| Approach: | They propose a multimodal entailment tree generator that promotes interpretable joint-modality reasoning by searching for trees of enanglement relationships between simple text-video evidence and higher-level conclusions that prove question-answer pairs. |
| Outcome: | The proposed approach performs on the TVQA benchmark and shows that it is state-of-the-art on full clips. |